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EXECUTIVE  SUMMARY 


OBJECTIVE  OF  THE  DEMONSTRATION 

This  project  evaluated  the  effectiveness  of  in-field  quality  control  (QC)  procedures  during  cued 
electromagnetic  induction  (EMI)  data  collection.  The  primary  objective  of  the  demonstration 
phase  of  this  project  was  to  gain  further  insight  into  the  field  practices  that  lead  to  the  most  effective 
and  efficient  cued  EMI  surveys.  Eor  this  demonstration  phase,  the  project  team  worked  with  a  field 
team  to  apply  an  in-field  QC  software  module  during  a  cued  survey  of  the  Eormer  WMA  to  identify 
anomalies  that  may  have  been  insufficiently  characterized  by  the  initial  data  collection.  After  the 
survey  and  final  ground  truth  stages  were  completed,  a  retrospective  analysis  of  the  WMA  data  set 
was  performed  to  identify  cases  where  recollects  based  on  the  in-field  decision  led  to  an 
improvement  in  data  quality  as  well  as  cases  where  the  recollect  was  unnecessary  (i.e.,  it  did  not 
provide  any  improvements  in  classification  features). 

It  should  also  be  noted  that  during  most  of  the  survey,  one  of  the  comer  receivers  for  the 
MetalMapper  software  was  malfunctioning.  It  is  unclear  to  what  extent  this  faulty  receiver 
influenced  the  in-field  decisions;  however,  it  is  possible  that  it  had  some  effect  on  the  accuracy  of 
in-field  target  location  estimates. 

TECHNOLOGY  DESCRIPTION 

The  in-field  QC  approach  includes  the  use  by  cued  sensor  operators  of  a  real-time  inversion 
software  module  that  provides  immediate  output  of  features  associated  with  each  anomaly 
investigated  by  cued  EMI  data  collection.  Among  the  relevant  features  provided  by  the  software 
is  an  estimate  of  the  location  of  the  buried  target.  If  the  lateral  offset  of  this  estimated  location  is 
>30  centimeters  (cm)  from  the  center  of  the  cued  sensor,  the  sensor  operator  can  reposition  the 
sensor  over  the  estimated  source  location  and  recollect  the  cued  data.  Visual  interpretation  of  the 
sensor  location,  the  estimated  target  location,  and  other  target  features  such  as  electromagnetic 
polarizabilities  is  enabled  by  the  in-field  QC  software. 

DEMONSTRATION  RESULTS 

During  the  field  demonstration  of  this  project,  the  Parsons  field  team  was  supplied  with  the  in¬ 
field  QC  software  during  their  cued  MetalMapper  survey  at  the  Eormer  Waikoloa  Maneuver  Area 
(WMA)  on  the  island  of  Hawaii.  During  this  survey,  the  field  team  encountered  1,032  unique 
anomaly  locations  with  the  MetalMapper.  Out  of  these  1,032  encounters,  231  resulted  in  recollects 
based  on  the  estimated  target  location  feedback  provided  by  the  in-field  QC  software. 

A  retrospective  analysis  was  performed  of  these  MetalMapper  data  to  determine: 

•  if  there  were  any  missed  recollect  opportunities,  i.e.,  cases  where  a  recollect  was  not 
performed,  but  should  have  been  performed; 

•  the  effectiveness  of  the  in-field  QC  process  by  quantifying  any  improvements  in  target 
features  obtained  by  recollecting  the  data;  and 

•  the  efficiency  of  the  in-field  QC  process  by  identifying  the  number  of  cases  where  the 
recollect  was  unnecessary,  i.e.,  it  did  not  produce  better  characterization  of  the  target. 
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These  reeolleet  statisties  were  used  to  develop  estimates  of  produetion  rates  for  surveys  eondueted 
using  the  in-field  QC  approaeh  and  for  surveys  where  no  in-field  reeolleet  deeision  is  made.  A 
summary  of  the  statisties  and  estimated  produetion  rates  are  as  follows: 

•  out  of  1,032  anomalies  investigated,  of  whieh  231  resulted  in  a  reeolleet,  there  was  1 
potential  missed  reeolleet  opportunity; 

•  out  of  231  reeolleets,  153  reeolleets  appeared  to  be  a  result  of  magnetie  geology  ereating 
false  souree  loeations; 

•  out  of  the  remaining  78  reeolleets  that  were  due  to  legitimate  sourees  (i.e.,  a  metal  objeet), 
46  resulted  in  improvements  in  target  eharaeterization;  and 

•  of  the  remaining  32  reeolleets  that  did  not  signifieantly  improve  target  eharaeterization,  1 1 
oases  were  found  where  the  unneoessary  reeolleet  may  have  been  avoided  with  the 
applioation  of  additional  quality  metrios  (i.e.,  in  addition  to  the  estimated  target  looation 
metrio). 

Estimated  produetion  rates  for  surveying  with  and  without  the  in-field  QC  prooess  were  23 
anomalies/hour  (hr)  (with  in-field  QC)  and  26  anomalies/hr  (without  in-field  QC). 

IMPLEMENTATION  ISSUES 

The  magnetie  geology  at  the  site  presented  the  most  signifioant  ohallenge  to  the  teohnology  and 
oontributed  to  the  lower-than-expeoted  produetion  rate  for  the  in-field  QC  approaeh.  The  46  oases 
that  resulted  in  quantifiable  improvements  in  target  features  are  an  example  of  the  potential 
benefits  of  applying  in-field  QC  to  oued  EMI  surveys.  Possible  ways  to  improve  the  effioienoy  of 
the  teohnology  at  ohallenging  sites,  suoh  as  the  WMA,  oould  inolude  improving  baokground 
seleotion  and  removal  during  in-field  QC  of  the  data,  or  implementing  multi-souroe  solvers  in  the 
in-field  inversion  to  aooount  for  magnetie  soil  effeots  and  high-target  densities. 
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1.0  INTRODUCTION 


This  project  was  undertaken  to  evaluate  and  implement  quality  control  (QC)  processes  for  cued 
electromagnetic  induction  (EMI)  data  collection  associated  with  munitions  classification  surveys. 
The  primary  objective  of  this  effort  was  to  establish  an  optimal  set  of  metrics  to  inform  field  teams 
of  the  classification  quality  of  the  data  collected  during  a  cued  EMI  survey.  Incorporating  these 
metrics  in  a  software  interface  will  enable  field  operators  to  determine  immediately  upon 
acquisition  if  cued  data  are  of  high  enough  quality  to  provide  useful  classification  features.  This 
ability  to  make  an  informed  in-field  decision  regarding  cued  EMI  data  quality  has  the  potential  to 
greatly  increase  the  efficiency  of  these  surveys.  Currently,  most  of  the  quality  analysis  for  cued 
EMI  surveys  is  performed  off-site  by  trained  analysts.  While  effective  for  identifying  instances  of 
poor  data  quality,  this  method  can  be  costly  as  it  often  requires  field  teams  to  redeploy  to  certain 
areas  to  reacquire  data.  Enabling  an  in-field  quality  decision  will  produce  immediate  corrective 
actions  that  will  obviate  the  need  for  subsequent  redeployment  and  reacquisition  by  the  field  team. 

1.1  BACKGROUND 

The  prevalence  of  innocuous  clutter  (e.g.,  scrap,  fragmentation)  at  Munitions  Response  (MR)  sites 
presents  a  challenge  to  remediation  efforts  that  often  devote  substantial  resources  to  the  excavation 
and  identification  of  these  non-hazardous  objects.  Over  the  last  5-10  years,  the  development  of 
advanced  EMI  sensor  arrays  that  enable  multi-axis  or  multi-angle  illumination  of  cued  anomalies 
has  enabled  the  implementation  of  methodologies  that  effectively  discriminate  clutter  from 
unexploded  ordnance  (UXO)  or  other  munitions  and  explosives  of  concern  (MEC).  These 
classification  methodologies  have  the  potential  to  significantly  improve  the  efficiency  of 
production  cleanup  efforts  by  reducing  the  time  and  costs  associated  with  removal  of  benign 
objects;  however,  collection  of  high-quality  data  with  advanced  sensors  is  critical  to  ensuring  the 
effectiveness  of  these  classification  algorithms. 

Currently,  the  standard  protocol  for  collecting  advanced  EMI  sensor  data  includes  static  data 
acquisition  over  anomalies  identified  (cued)  by  prior  dynamic  or  digital  geophysical  mapping 
(DGM)  surveys.  These  data  are  typically  analyzed  by  off-site  geophysicists  upon  completion  of 
the  cued  survey.  Often,  this  analysis  results  in  a  number  of  anomalies  that  are  found  to  be 
insufficiently  characterized  by  the  initial  cued  data  collection  as  a  result  of  sub-optimal  placement 
of  the  EMI  sensor  during  the  survey.  Subsequently,  reacquisition  of  data  is  necessary  and 
additional  surveying  is  required.  Methods  that  significantly  reduce  reacquisitions  and  improve  the 
overall  quality  of  the  data  before  completion  of  the  initial  cued  survey  will  significantly  improve 
the  efficiency  of  production  operations. 

Providing  the  field  team  with  the  capability  to  assess  data  quality  immediately  following  the  initial 
data  collection  could  significantly  improve  the  cued  survey  process.  By  removing  the  off-site 
analyst  from  the  initial  quality  decision-making,  the  field  team  can  take  immediate  action  to 
reposition  and  recollect  certain  targets  if  deemed  necessary.  Immediate  recollection  based  on  in¬ 
field  QC  can  be  much  more  efficient  than  subsequent  reacquisition  based  on  off-site  decision¬ 
making.  Off-site  analysis  may  still  be  important  for  identifying  complicated  target  scenarios  (e.g., 
multi-object,  magnetic  geology);  however,  many  of  the  reacquires  due  to  common  errors  such  as 
inaccurate  target  picking  could  be  replaced  by  in-field  decision-based  recollects. 
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1.2  OBJECTIVES  OF  THE  DEMONSTRATION 


The  primary  objective  of  the  demonstration  was  to  gain  further  insight  into  the  field  practices  that 
lead  to  the  most  effective  and  efficient  cued  EMI  surveys.  For  the  demonstration,  the  project  team 
worked  with  a  field  team  to  apply  an  in-field  QC  software  module  during  a  cued  survey  of  the 
Former  WMA  to  identify  anomalies  that  may  have  been  insufficiently  characterized  by  the  initial 
data  collection.  After  the  survey  and  final  ground  truth  stages  were  completed,  a  retrospective 
analysis  of  the  WMA  data  set  was  performed  to  identify  cases  where  recollects  based  on  the  in¬ 
field  decision  led  to  an  improvement  in  data  quality  as  well  as  cases  where  the  recollect  was 
unnecessary  (i.e.,  the  recollect  did  not  provide  any  improvements  in  classification  features). 

By  testing  the  in-field  QC  process  in  a  live  site  venue,  this  demonstration  allowed  the 
quantification  of  the  benefits  of  the  approach  as  well  as  identification  of  practical  strategies  for 
improving  the  utility  and  effectiveness  of  the  approach.  The  Former  WMA  site  presented  a  unique 
set  of  challenges  associated  with  magnetic  soils;  however,  the  suggested  improvements  to  the  in¬ 
field  QC  process  can  be  applied  generally  to  sites  where  background  soil  response  may  be  variable. 

1.3  REGULATORY  DRIVERS 

Military  Munitions  Response  Program  (MMRP)  regulations  require  well-defined  Standard 
Operating  Procedures  (SOPs)  for  DGM  surveys  conducted  throughout  MR  sites.  These  SOPs 
include  guidelines  for  conducting  daily  verification  of  instrument  functionality  for  both 
geophysical  and  positioning  equipment  used  during  the  survey.  Procedures  are  also  defined  for  the 
processing  and  analysis  of  geophysical  data  to  ensure  that  the  survey  produces  high-quality  data 
that  will  indicate  the  location  of  any  potential  contaminants.  While  the  SOPs  associated  with  DGM 
operations  have  been  refined  through  years  of  practice  to  produce  effective  survey  results,  the 
introduction  of  cued  classification  surveys  to  MMRP  projects  will  require  significant 
modifications  to  these  procedures. 

The  in-field  QC  module  provides  the  field  team  with  an  intuitive  means  for  assessing  not  just  basic 
instrument  functionality,  but  also  classification  quality  of  the  data.  Without  this  immediate 
feedback,  the  field  team  must  rely  on  off-site  analysis  before  knowing  if  data  quality  objectives 
(DQOs)  are  achieved.  Thus,  providing  the  field  team  with  relevant  quality  metrics  is  extremely 
important  for  ensuring  that  classification  surveys  are  conducted  in  an  efficient  manner  and  meet 
the  DQOs  that  are  specific  to  these  cued  surveys. 
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2.0  TECHNOLOGY 


The  basis  for  the  QC  module  technology  is  a  C++  inversion  and  classification  Application 
Programming  Interface  (API)  that  provides  access  to  physical  model  (i.e.,  dipole)  inversion 
algorithms,  which  can  be  used  to  rapidly  estimate  a  variety  of  model  parameters  associated  with 
each  anomaly.  The  QC  module  allows  the  field  team  to  obtain  information  about  target  physical 
properties  immediately  after  acquiring  a  cued  EMI  data  file.  This  information  can  then  be  used  to 
assess  the  quality  of  the  data  based  on  the  reliability  of  these  target  features. 

2.1  TECHNOLOGY  DESCRIPTION 

The  QC  module  functions  as  a  parallel  process  to  the  cued  EMI  sensor  data  acquisition  software. 
As  such,  the  QC  module  does  not  provide  any  data  acquisition  functionality;  it  is  used  solely  for 
immediate  in-field  analysis  of  the  cued  sensor  data.  Existing  sensor  data  acquisition  software  (e.g., 
MetalMapper  EMSDAcquire)  provides  the  operator  with  the  appropriate  interfaces  to  set  the 
acquisition  parameters  and  trigger  an  acquisition  event;  however,  all  other  processes  associated 
with  the  survey  including  sensor  navigation  and  data  quality  analysis  are  facilitated  by  the  QC 
module,  hollowing  the  acquisition  of  a  cued  sensor  data  fide,  the  QC  module  will  immediately  load 
this  file  and  invert  the  file  data.  The  output  of  these  inversions  is  subsequently  transferred  to  the 
operator  navigation  and  QC  interfaces  that  display  the  various  recovered  dipole-model  parameters 
(e.g.,  polarizabilities,  location,  depth)  as  well  as  the  raw  data  channel  output.  Eigure  lError! 
Reference  source  not  found,  shows  a  screenshot  capturing  a  typical  arrangement  of  these 
interfaces.  It  is  possible  to  make  both  the  data  acquisition  and  QC  functions  available  within  the 
same  display. 
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Figure  I.  Typical  Operator  Display  Setup  Showing  QC  and  Acquisition  Interfaces. 
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The  QC  module  provides  the  main  navigation  and  analysis  interfaces,  while  EM3D  provides  the 
acquisition  capability  with  a  simple  acquire  button. 


The  QC  module  navigation  interface  facilitates  sensor  positioning  and  data  recollects  by  displaying 
target  coordinates  relative  to  the  cued  EMI  sensor  head  location.  Target  picks  identified  from 
analysis  of  the  DGM  data  can  be  loaded  into  the  navigation  interface  to  show  the  location  of  each 
anomaly  that  will  be  interrogated  with  the  cued  EMI  sensor.  Once  the  operator  selects  an  anomaly 
for  interrogation,  the  navigation  interface  will  display  both  the  anomaly  location  and  the  sensor 
head  location  (Eigure  2).  As  the  operator  maneuvers  the  sensor  head  closer  to  the  anomaly,  the 
navigation  display  auto-zooms  to  enable  accurate  positioning  of  the  sensor  over  the  anomaly.  The 
display  provides  a  sensor  frame-of-reference  view  such  that  the  sensor  heading  is  always  directed 
toward  the  top  of  the  operator  display  screen.  Once  the  sensor  head  is  positioned  over  the  anomaly 
coordinates,  the  operator  acquires  a  data  file.  Immediately,  the  data  are  inverted  and  the  estimated 
coordinates  of  the  target  are  displayed.  If  necessary,  the  operator  can  then  reposition  the  sensor 
using  these  new  coordinates. 


Figure  2.  QC  Module  Navigation  Interface. 

LEFT:  The  Navigation  Interface  Displays  the  Anomaly  Coordinates  and  the  Sensor  Location.  RIGHT: 
the  Operator  Moves  the  Sensor  Closer  to  an  Anomaly,  the  Display  Performs  an  Auto-Zoom  to  Facilitate 
Accurate  Positioning  of  the  Sensor  Head  Over  the  Anomaly.  After  the  data  are  acquired,  the  QC  module 
updates  the  navigation  display  with  the  estimated  target  location  (pink  dot). 


In  addition  to  providing  the  operator  with  the  estimated  target  location,  the  QC  module  also 
provides  information  about  model  parameters  that  correspond  to  target  features.  Classification 
features,  such  as  target  polarizabilities,  are  compared  to  those  catalogued  in  a  target  library.  This 
library  corresponds  to  possible  targets  of  interest  (TOIs)  that  may  be  located  at  the  site.  The  TOI 
library  is  built  from  data  sets  collected  in  controlled  calibration  areas  within  the  site  (Eigure  3). 
Cued  EMI  data  are  collected  in  a  calibration  area  (i.e.,  a  test  pit)  over  TOIs  that  are  indigenous  to 
the  site.  Eeatures  from  these  TOIs  are  then  added  to  the  library.  During  the  subsequent  cued  survey, 
the  QC  interface  will  display  the  polarizabilities  corresponding  to  each  anomaly  item  and  compare 
them  to  the  library  features.  The  QC  interface  also  indicates  the  best  match  TOI.  The  library  can 
be  expanded  with  additional  data  sets  collected  throughout  the  survey  if  additional  items  of  interest 
are  found. 
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Figure  3.  Target  Libraries  and  Features. 

LEFT:  Target  Libraries  Are  Built  from  Data  Collected  in  Controlled  Calibration  Areas  within  the  Site. 
This  Image  Shows  a  37-mm  Projectile  Placed  in  a  Test  Pit  for  Library  Data  Collection.  RIGHT:  Library 
Features  Are  Subsequently  Loaded  Into  the  QC  Module.  The  QC  interface  displays  polarizabilities 
recovered  from  cued  EMI  data  inversion  and  compares  them  to  the  best-match  library  features. 


2.2  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGY 

The  QC  module  provides  several  advantages  for  field  teams  conducting  cued  EMI  surveys. 
Currently,  there  are  no  commercially  available  software  packages  that  directly  link  with  advanced 
sensor  output  to  provide  data  inversion  results  in  real-time  (i.e.,  immediately  after  the  acquisition 
of  a  data  file).  The  capability  for  sensor  operators  to  obtain  inversion  parameters,  such  as  estimated 
target  location  and  target  polarizabilities  while  the  sensor  is  still  in  proximity  to  the  target  pick 
location,  has  significant  implications  for  production  rates.  By  providing  the  operator  with  the 
necessary  information  to  assess  data  quality,  the  QC  module  facilitates  immediate  corrective  action 
based  on  in-field  decisions. 

The  primary  limitation  of  the  technology  that  became  apparent  during  demonstration  of  the 
technology  at  the  Former  WMA  is  the  interference  of  magnetic  soils  with  the  real-time  inversion 
output.  Retrospective  analysis  of  the  Waikoloa  data  indicated  that  in  many  instances,  the  in-field 
inversion  produced  source  locations  due  to  geology.  Mitigating  the  effects  of  magnetic  soils  poses 
a  significant  challenge  not  just  for  in-field  analysis,  but  for  off-site  analysts  as  well.  It  may  be 
difficult  to  remove  these  effects  entirely  during  the  in-field  analysis;  however,  possible  solutions 
could  include  more  frequent  background  updates  to  the  in-field  software,  the  implementation  of  a 
decay  fit  to  indicate  to  the  field  operator  if  the  response  is  likely  due  to  ground  conditions,  or  the 
implementation  of  a  multi-object  inversion  that  accounts  for  the  ground  response  in  addition  to  the 
target  response. 
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3.0  PERFORMANCE  OBJECTIVES 

The  basis  for  the  field  demonstration  eonsisted  of  a  eued  (static)  MetalMapper  survey  of  anomalies 
identified  from  a  prior  EM-61  DGM  survey.  The  MetalMapper  field  team  conducted  a  survey  of 
approximately  1 ,000  anomalies  while  using  the  in-field  QC  module.  The  team  conducted  recollects 
for  anomaly  locations  where  the  in-field  software  indicated  the  presence  of  a  source  that  was  more 
than  30  centimeters  (cm)  from  the  center  of  the  MetalMapper.  These  procedures  led  to  231 
recollects  over  the  course  of  the  survey.  These  recollects  formed  the  basis  for  the  retrospective 
analysis  of  the  QC  performance.  With  the  incorporation  of  the  QC  module  into  the  survey  protocol, 
there  were  five  possible  scenarios  that  could  be  used  to  establish  the  capabilities  and  limitations  of 
the  technology.  These  scenarios  are  as  follows: 

1 .  Cases  where  no  recollect  was  taken  and  no  recollect  was  necessary.  These  cases  accounted 
for  about  80%  of  the  total  survey.  While  these  instances  did  not  contribute  to  any 
quantitative  assessment  of  the  technology  performance,  they  can  be  used  as  a  baseline  to 
assess  the  qualitative  aspects  associated  with  ease  of  use  and  production  efficiency. 

2.  Cases  where  no  recollect  was  taken,  but  was  necessary.  Such  instances  are  representative 
of  the  technology  limitations  (i.e.,  cases  where  the  in-field  QC  process  failed  to  correctly 
identify  a  recollect  opportunity).  Out  of  1,032  anomalies,  one  case  was  identified  where 
repositioning  and  recollecting  may  have  been  useful.  A  possible  missed  recollect 
opportunity  occurred  for  seed  item  WK-1047,  where  the  sole  MetalMapper  acquisition  was 
collected  at  an  offset  of  43  cm  from  the  seed  location.  Seed  WK-1047  was  one  of  the  two 
seeds  missed  in  the  classification  stage  by  the  project  analysts.  While  it  is  unclear  if 
repositioning  and  recollecting  would  have  improved  classification  features,  a  43-cm  offset 
would  normally  produce  a  recollect  decision.  Most  likely,  this  recollect  opportunity  was 
missed  because  of  the  very  strong  soil  response  in  this  area  (post-survey  analysis  of  this 
anomaly  produced  ground-like  features),  which  may  have  produced  inaccurate  in-field 
source  estimates. 

3.  Cases  where  a  recollect  was  taken,  was  necessary,  and  improved  the  classification 
features.  This  scenario  enables  a  quantitative  assessment  of  the  technology  performance 
by  providing  clear  examples  of  the  advantages  afforded  by  the  in-field  QC  process.  The 
analysis  indicated  that  there  were  46  cases  where  the  recollect  improved  classification 
features  associated  with  the  anomaly  (in  some  cases  this  improvement  was  significant 
enough  that  without  the  recollect,  correct  classification  may  have  been  difficult). 

4.  Cases  where  the  in-field  model  did  not  provide  accurate  characterization  of  the  survey 
space.  This  scenario  could  include,  for  example,  multi-object  cases  where  the  QC  module 
correctly  identifies  a  problematic  target,  but  does  not  provide  the  in-field  capability  to 
characterize  the  situation  correctly;  however,  for  the  WMA  survey,  the  greatest  challenge 
to  the  in-field  QC  models  was  presented  by  the  magnetic  geology.  Out  of  the  23 1  recollects 
conducted  in  the  WMA  survey,  153  of  these  appeared  to  be  caused  by  the  in-field  models 
indicating  a  ground  source  (1 14  of  these  occurred  in  areas  where  no  target  was  present,  39 
occurred  in  areas  where  small  debris  was  found). 
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5.  Cases  where  a  recollect  was  taken,  but  was  unnecessary.  In  other  words,  this  scenario 
comprises  cases  where  the  initial  acquisition  provided  sufficient  classification  features. 
During  retrospective  analysis,  32  cases  were  found  where  the  initial  acquisition  produced 
accurate  features  for  classification  and  the  recollect  did  not  offer  any  significant 
improvements.  These  were  cases  where  the  30-cm  recollect  rule  was  likely  overly 
conservative. 

Each  of  the  aforementioned  scenarios  is  associated  with  specific  performance  objectives.  During 
the  post-survey  analysis,  a  set  of  metrics  was  applied  corresponding  to  each  of  these  objectives. 
Table  1  summarizes  the  performance  objectives  and  corresponding  metrics  used  during 
retrospective  analysis. 


Table  1.  Performance  Objectives 


Performance 

Objective 

Metric 

Data  Required 

Success  Criteria 

Quantitative  Performance  Objectives 

Identification  of  all 

recollect 

opportunities 

Percent  of  reacquires  out  of 
total  recollect  opportunities 

•  Off-site  QC  analysis  of 
all  MM  initial 
acquisition  fdes 

Preac<0.1 

Effective  corrective 
action 

Percent  recollects  out  of 
total  recollects  resulting  in 
improved  target  features 

•  Off-site  QC  analysis  of 
MM  recollects  and 
corresponding  initial 
acq.  files 

Prec(I)>0.9 

Effective  in-field 
characterization 

Number  of  ineffective 
recollects  due  to  inadequate 
in-field  models 

•  MM  recollect  files  that 
did  not  provide 
improved  features 

Insufficient  model 
characterization  <5%  of 
total  recollect  cases 

Effective  quality 
metrics 

Number  of  ineffective 
recollects  eliminated 

•  MM  recollect  fdes  that 
did  not  provide 
improved  features 

Any  reduction  in 
ineffective  recollects 

Production  rate 

Number  of  unique 
anomalies  surveyed  per  day 
(in  terms  of  hourly  quotas) 

•  Field  logs,  data  file  time 
stamps 

Site  dependent 

Qualitative  Performance  Objectives 

Ease  of  use 

•  Operator  feedback 
regarding  intuitiveness 
of  display,  QC  results, 
and  process  flows 

MM  -  MetalMapper,  Preac  -percentage  of  reacquisitions,  Prec(I)  -  percentage  of  recollects  resulting  in  improved  classification 
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4.0  SITE  DESCRIPTION 


MetalMapper  data  collected  at  the  Former  WMA  provide  the  basis  for  the  technology 
demonstration.  This  area  comprises  approximately  100,000  acres  on  the  northwest  portion  of  the 
island  of  Hawaii  and  includes  property  that  was  used  as  a  military  training  camp  and  artillery  range 
(Figure  4).  Over  100  munitions  types  including  mortars,  projectiles,  hand  grenades,  and  rockets 
are  known  to  be  present  at  the  site.  Details  regarding  site  history  and  land  usage  can  be  found  on 
the  U.S.  Army  Corps  of  Engineers  (USAGE)  Honolulu  District  website.^ 


Figure  4.  Former  WMA  (from  USAGE  Waikoloa  Map). 

The  arid  site  contains  sparse  vegetation  making  it  suitable  for  vehicle  surveys  (Figure  5).  The  main 
deployment  challenge  is  the  rocky  soil,  which  can  damage  tires  and  make  transport  of  equipment 
across  the  site  difficult.  The  site  is  also  known  to  contain  significant  magnetic  geology,  which  can 
be  problematic  for  electromagnetic  background  removal  during  data  processing. 


'  http://www.poh.usace.army.mil/Missions/Environmental/FUDSAVaikoloa.aspx. 
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Figure  5.  Former  WMA  Site  Photos. 


MetalMapper  data  were  eollected  within  three  areas  of  interest  (AOIs),  whieh  ineluded  task  order 
(TO)17  and  TO20  areas  A  and  B.  Eaeh  of  these  areas  provided  unique  magnetic  geology  that 
created  variable  levels  of  electromagnetic  background  across  the  dataset.  The  magnetic  soil 
influence  proved  to  be  one  of  the  most  significant  challenges  in  the  implementation  of 
classification  and  in-field  QC  at  this  site. 
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5.0  TEST  DESIGN 


Demonstration  of  the  in-field  QC  approach  was  included  as  a  supplement  to  ongoing  classification 
studies  conducted  at  the  WMA.  The  unique  geology  of  the  WMA  offered  significant  challenges  to 
classification  and,  therefore,  this  site  was  selected  for  demonstration  of  several  Environmental 
Security  Technology  Certification  Program  (ESTCP)  technologies.  The  project  team  coordinated 
with  Parsons,  who  were  conducting  one  of  the  classification  technology  demonstrations  at  this  site, 
to  provide  the  field  team  a  version  of  the  in-field  QC  module  to  use  during  the  cued  MetalMapper 
survey  of  the  aforementioned  three  AOIs.  Parsons  field  personnel  were  familiar  with  or  had  prior 
experience  using  the  in-field  QC  software,  so  transition  of  the  technology  was  relatively 
straightforward. 

The  objective  for  conducting  this  demonstration  was  to  obtain  data  about  the  utility  of  the  in-field 
QC  process  and  at  the  same  time  to  hopefully  provide  the  Parsons  team  with  a  technology  that 
would  facilitate  the  technology  demonstration.  A  comprehensive  summary  of  details  related  to  the 
field  activities  (e.g.,  system  calibration,  data  collection,  ground  truth  validation)  for  this 
demonstration  can  be  found  in  the  Parsons  MR-201 104  demonstration  report  (Van  et  ah,  2015). 
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6.0  DATA  ANALYSIS  AND  PRODUCTS 


The  analysis  focused  on  retrospective  interpretation  of  classification  results  to  determine  the 
effectiveness  of  the  in-field  QC  process.  To  obtain  relevant  data  and  model  parameters  from  each 
MetalMapper  anomaly,  dipole  inversions  (single  and  multi-object)  were  performed  for  each  static 
MetalMapper  file.  These  inversions  provided  parameters  such  as  polarizabilities,  estimated 
location,  and  fit  that  were  subsequently  used  to  obtain  metrics  for  each  anomaly.  The  project  team 
also  coordinated  with  analysts  from  Dartmouth  College  (Shubitidze,  2015),  who  performed 
primary  classification  analysis  for  the  WMA  site,  to  identify  any  specific  anomalies  that  might 
further  elucidate  the  benefits  or  shortcomings  of  the  in-field  QC  method. 

6.1  PARAMETER  ESTIMATION 

To  obtain  the  relevant  parameters  that  would  quantify  any  improvements  in  recollected  data  files, 
dipole  inversions  were  performed  for  each  MetalMapper  file.  One  of  the  most  important  steps  in 
this  processing  stage  was  the  selection  of  background  data  that  would  be  used  for  background 
removal  in  each  cued  data  file.  Because  background  response  was  so  variable  across  the  WMA, 
selection  of  different  backgrounds  could  produce  very  different  sets  of  model  parameters  obtained 
from  each  inversion.  Background  files  were  initially  used  that  had  timestamps  closest  to  those  of 
the  associated  cued  files.  If  background  removal  using  these  files  still  left  a  significant  background 
response  in  the  data,  other  background  fdes  collected  on  the  given  day  would  be  tried  to  see  if  any 
improvements  in  background  removal  could  be  realized.  In  some  cases,  it  was  only  possible  to 
obtain  parameters  associated  with  a  background  response  (this  was  the  case,  for  example,  with 
what  turned  out  to  be  “no-contact”  sources). 

After  background  removal,  each  static  file  was  inverted  using  both  single  and  multi-object  dipole 
models.  In  some  cases,  the  multi-object  could  be  used  to  constrain  the  ground  response.  The 
polarizabilities,  estimated  source  location,  and  data/model  fit  for  each  inversion  were  saved  for 
further  review  of  performance  metrics. 

6.2  PERFORMANCE  ANALYSIS 

Performance  analysis  included  a  review  of  the  aforementioned  parameters  associated  with  each 
recollect  anomaly  as  well  as  identification  of  specific  anomalies  that  proved  challenging  for  the 
final  classification  stage  (i.e.,  based  on  retrospective  receiver  operating  characteristic  [ROC]  curve 
analysis).  The  objective  metrics  were  applied  to  each  of  these  cases. 

6.2.1  Objective:  Identification  of  All  Recollect  Opportunities 

Any  cued  sounding  based  on  a  DGM  pick  could  be  considered  a  recollect  opportunity;  however,  one 
of  the  biggest  issues  with  identifying  these  opportunities  was  whether  to  include  the  large  number  of 
no-contact  encounters  (i.e.,  no  target  found  during  excavation)  as  a  potential  recollect  opportunity. 
There  were  a  large  number  of  anomalies  (about  half  of  all  the  cued  encounters)  that  could  not  be 
attributed  to  a  legitimate  source  (other  than  the  ground  or  “hot  rocks”).  Therefore,  these  encounters 
would  have  a  significant  impact  on  the  overall  recollect  statistics.  Because  these  encounters  represented 
an  opportunity  to  make  a  “no  recollecf  ’  decision,  they  were  included  in  the  analysis;  however,  it  should 
be  noted  that  the  in-field  QC  module  did  not  have  specific  models  to  reject  ground  response,  therefore, 
such  cases  were  expected  to  be  challenging  for  the  in-field  decision. 
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6.2.2  Objective:  Effective  Corrective  Action 

To  determine  whether  effective  corrective  action  was  taken,  the  recovered  model  parameters  from 
each  recollect  were  compared  to  those  from  the  initial  acquisition.  For  recollects  associated  with 
TOI,  this  analysis  applied  a  library  fit  to  the  polarizabilities  to  determine  any  improvements  in 
target  features.  For  non-TOI  encounters,  a  set  of  model  parameters  were  applied  related  to 
data/model  fit,  polarizability  consistency  between  successive  measurements,  and  polarizability 
noise  to  determine  if  the  recollect  provided  improvements  in  features.  Finally,  for  cases  where 
strong  ground  response  was  observed,  the  polarizability  decay  was  assessed  to  determine  whether 
the  features  were  associated  with  a  metal  object  or  the  ground  response. 

6.2.3  Objective:  Effective  In-Field  Characterization 

For  each  recollect  case  that  did  not  lead  to  improved  target  features,  the  reason  that  classification 
quality  was  not  improved  was  identified.  Because  of  the  interest  particularly  in  cases  where  the  in¬ 
field  models  did  not  accurately  characterize  the  target  space,  data/model  fits  and  ground  truth 
records  were  evaluated  to  determine  if,  for  example,  multiple  targets  in  the  area  may  have  led  to 
misguided  placement  of  the  sensor.  The  polarizability  decay  was  also  assessed  to  determine  if 
ground  response  likely  had  any  significant  influence  on  the  in-field  decision. 

6.2.4  Objective:  Effective  Quality  Metrics 

For  recollect  cases  where  the  initial  acquisition  provided  sufficient  characterization  of  the 
anomaly,  the  application  of  alternative  metrics  to  support  a  recollect  decision  was  evaluated.  Also 
evaluated  was  whether  the  positioning  requirements  (30  cm  or  closer)  for  the  initial  acquisition 
were  overly  stringent  and  resulted  in  too  many  recollects.  The  goal  was  to  determine  if  these 
unnecessary  recollects  could  be  avoided  without  risking  any  reduction  in  data  quality. 

6.2.5  Objective:  Production  Rate 

To  establish  any  efficiency  gains  resulting  from  the  in-field  QC  process,  the  production  rates 
corresponding  to  the  in-field  QC  protocol  and  the  standard  data  collection  protocol  (no  in-field 
QC)  were  evaluated  based  on  time  stamps  recorded  in  the  data  files.  For  example,  the  data  time 
stamps  in  several  groups  of  consecutively  recorded  MetalMapper  files  were  used  to  establish  the 
average,  ideal  production  rates  for  both  data  collection  methods.  Basing  this  analysis  on  the  time 
stamp  information  ensured  that  production  rate  estimates  were  not  influenced  by  inadvertent  down 
time  (e.g.,  for  system  transport  across  the  site  or  hardware  troubleshooting)  that  would  be  atypical 
of  standard  MetalMapper  surveys. 


14 


7.0  PERFORMANCE  ASSESSMENT 


The  following  is  a  summary  of  the  recollect  statistics  for  the  WMA  MetalMapper  survey: 

•  Out  of  1 ,032  total  anomaly  encounters,  there  were  23 1  unique  recollects; 

•  Out  of  104  TOI  encounters,  there  were  18  unique  recollects; 

•  Out  of  393  non-TOI  (debris)  encounters,  there  were  99  unique  recollects;  and 

•  Out  of  535  no-contact  (no  target)  encounters,  there  were  1 14  unique  recollects. 

These  statistics  provide  the  basis  for  the  analysis  of  the  performance  objectives. 

7.1  OBJECTIVE:  IDENTIFICATION  OF  ALL  RECOLLECT  OPPORTUNITIES 

To  determine  if  there  were  any  missed  recollect  opportunities  during  the  WMA  survey,  cases 
where  the  final  MetalMapper  location  was  >30  cm  from  an  excavated  source  were  identified.  Three 
potential  cases  were  found,  as  discussed  here. 

The  first  case  involved  anomaly  WK-73 — a  37-millimeter  (mm)  seed  buried  at  1 1  cm  depth.  In 
this  case,  the  initial  (and  only)  MetalMapper  file  associated  with  this  anomaly  was  acquired  at  an 
offset  of  75  cm.  This  anomaly  was  problematic  for  analysts  because  the  only  features  obtained 
were  associated  with  a  ground  response  due  to  the  large  offset.  While  repositioning  over  the  seed 
location  would  undoubtedly  have  provided  better  data  for  characterizing  the  anomaly,  it  was  not 
thought  that  this  was  an  instance  where  the  in-field  QC  failed.  A  75 -cm  offset  placed  the  seed  well 
outside  the  footprint  of  the  sensor.  Under  ideal  conditions  (i.e.,  no  significant  soil  response  or 
presence  of  other  nearby  anomalies),  it  is  possible  that  the  in-field  model  could  have  located  a 
source  this  far  from  the  sensor  and  provided  some  guidance  for  better  positioning;  however,  given 
the  challenges  posed  by  field  conditions,  accurate  features  would  not  typically  be  expected  to  be 
obtained  from  a  source  located  this  far  from  the  sensor.  This  case  represented  an  error  in  the  initial 
positioning  of  the  sensor  and  was  therefore  removed  from  the  list  for  classification  analysis. 

The  other  two  cases  involved  anomalies  WK-1027  and  WK-1047,  which  were  a  small  Industry 
Standard  Object  (ISO)  seed  and  a  60-mm  seed,  respectively.  These  two  seeds  proved  problematic 
for  the  classification  analysts  as  evidenced  by  the  ROC  curve  shown  in  Figure  6. 
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Figure  6.  Independent  Scoring  Results  for  Dartmouth  College  WMA  MetalMapper 
Analysis  Indicate  Two  Difficult  Targets:  WK-I027  and  WK-I047. 

In  both  of  these  eases,  the  features  obtained  during  analysis  appeared  to  be  dominated  or  degraded 
by  the  ground  response.  Thus,  insufficient  background  removal  appeared  to  be  the  major  cause  of 
the  classification  failure;  however,  the  project  team  wanted  to  determine  if  the  sensor  positioning 
could  have  been  a  factor  as  well.  Since  neither  of  these  cases  included  a  recollect,  whether  the 
initial  sensor  location  may  have  contributed  to  the  incorrect  classification  decision  was  assessed. 

For  seed  WK-1027,  the  sensor  was  located  27  cm  from  the  target.  Since  this  offset  is  within  the 
conservative  30-cm  objective  radius,  this  case  was  not  considered  to  be  a  failure  of  the  in-field  QC 
model.  It  is  possible  that  given  the  small  size  of  the  object  and  the  dominant  ground  response, 
positioning  the  center  of  the  sensor  closer  to  the  object  could  have  provided  better  results  in  this 
case;  however,  it  is  believed  that  this  case  is  more  representative  of  the  challenges  posed  by 
magnetic  soils  than  of  the  limitations  of  the  in-field  QC  approach. 

For  seed  WK-1047,  the  sensor  was  located  43  cm  from  the  target.  This  offset  is  significantly 
outside  the  30-cm  objective  radius  and,  therefore,  this  is  considered  to  be  a  missed  recollect 
opportunity.  While  it  is  likely  that  under  ideal  conditions  (i.e.,  no  significant  soil  response  or 
presence  of  other  nearby  anomalies)  it  would  be  possible  to  obtain  accurate  classification  features 
at  a  43-cm  offset,  the  presence  of  a  significant  ground  response  makes  it  very  difficult  to  obtain 
representative  features  with  the  sensor  at  this  location.  Most  likely,  if  the  in-field  QC  model  had 
access  to  a  representative  background  at  this  location,  a  better  QC  decision  could  have  been  made 
at  the  time. 

To  obtain  the  metric  for  this  objective,  the  number  of  remaining  recollect  opportunities  (1,032  - 
231  =  801)  were  identified.  Out  of  801  possible  additional  recollect  opportunities,  it  is  believed 
that  there  is  only  one  case  where  repositioning  the  sensor  may  have  led  to  a  better  classification 
result.  Therefore,  the  percentage  of  reacquisitions  (Preac)  <0.1  was  achieved. 
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1.1  OBJECTIVE:  EFFECTIVE  CORRECTIVE  ACTION 


To  determine  whether  the  recolleets  provided  signifieant  improvements  in  the  classifieation  quality 
of  the  data,  the  features  obtained  from  eaeh  recolleet  were  analyzed  and  compared  to  those  of  the 
associated  initial  acquisition.  For  recollects  over  TOI,  improvements  were  quantified  by  assessing 
the  polarizability  fit  to  the  relevant  TOI  library.  Figure  7  shows  an  example  of  an  effective  recollect 
over  a  37-mm  seed. 


Figure  7.  Polarizabilities  Obtained  from  Two  Different  Acquisitions  Over  WK-724, 

a  37-mm  Seed. 

For  the  initial  acquisition,  the  sensor  was  offset  34  cm  from  the  target.  The  recollect  placed  the  sensor  at 
an  offset  of  13  cm  from  the  target.  The  polarizabilities  (blue,  red,  and  green  lines)  show  a  much  better 
match  to  the  library  (dark  grey  lines)  in  the  recollect. 

The  results  shown  in  Figure  7  reflect  a  good  example  of  improvements  achieved  by  recollecting 
with  better  sensor  positioning  (13  cm  lateral  offset  in  the  recollect  compared  to  34  cm  lateral  offset 
in  the  initial  acquisition).  The  initial  acquisition  did  not  enable  accurate  recovery  of  the 
polarizabilities;  however,  the  recollect  provided  features  that  matched  closely  to  the  library. 
Review  of  the  data  showed  that  the  initial  acquisition  may  have  been  influenced  by  a  small,  nearby 
piece  of  clutter  that  was  less  evident  in  the  recollect  location. 

For  recollects  over  non-TOI  (i.e.,  frag  or  other  debris),  any  significant  improvements  using  model 
parameters  derived  from  the  inversion  of  the  data  were  quantified.  For  example.  Figure  8  compares 
the  polarizabilities  obtained  from  initial  and  recollect  acquisitions  over  munition  debris  (WK- 
1053,  fuze  clip). 
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Figure  8.  Polarizabilities  Obtained  from  Two  Different  Acquisitions  Over  WK-1053,  a 

Piece  of  Munition  Debris. 

The  recollect  provides  better  constraint  on  the  third  polarizability  (green  line). 

The  results  shown  in  Figure  8  were  quantified  using  model  noise  and  model  consistency  metrics. 
Further  validation  was  later  provided  by  library  matching  to  polarizabilities  obtained  from  other 
fuze  clips  encountered  throughout  the  site.  The  recollect  shown  in  Figure  8  provided  features  that 
were  much  more  consistent  with  those  from  other  encounters  with  this  target,  which  turned  out  to 
be  a  common  debris  item  at  the  site. 

Over  the  course  of  this  analysis,  several  encounters  were  discovered  where  the  recollect  provided 
an  entirely  different  set  of  features  from  those  of  the  initial  acquisition.  These  cases  occurred  when 
the  initial  acquisition  was  dominated  by  the  ground  response  and  the  recollect  was  able  to  resolve 
the  features  associated  with  the  actual  target.  A  visual  QC  check  of  the  polarizabilities  could 
identify  these  cases  as  shown  in  Figure  9. 


Figure  9.  Example  of  the  Recollect  Providing  Better  Resolution  of  Target  Features. 

In  this  case,  the  initial  acquisition  produced  polarizabilities  dominated  by  the  ground  response.  The 
recollect  provides  features  that  more  accurately  represent  the  actual  target  (a  piece  of  frag). 
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Overall,  46  cases  were  found  where  the  recollect  provided  significant  improvements  in  target 
features.  While  this  number  is  a  minority  of  the  231  total  recollects,  it  should  be  noted  that  most 
of  the  recollects  (153  cases)  appeared  to  be  due  to  inaccurate  source  locations  created  by  the  ground 
response.  Counting  these  ground  recollects,  a  percentage  of  recollects  resulting  in  improved 
classification  (Prec(I))  =  0.19  is  obtained  (i.e.,  46  out  of  231  recollects  resulted  in  improved 
features).  Disregarding  the  ground  recollects,  a  Prec(I)  =  0.59  is  obtained  (i.e.,  46  out  of  78 
recollects  provided  improvements).  Both  of  these  values  fall  short  of  the  objective  Prec(I)  =  0.9 
(i.e.,  90%  of  recollects  result  in  improved  target  features). 

7.3  OBJECTIVE:  EFFECTIVE  IN-FIELD  CHARACTERIZATION 

The  majority  of  cases  where  the  recollect  did  not  lead  to  improved  target  features  appeared  to  be 
a  result  of  insufficient  background  removal  or  inadequate  ground  models.  An  example  of  this  is 
provided  in  Figure  10,  which  shows  the  initial  acquisition  and  the  recollect  over  anomaly  WK-68, 
a  piece  of  frag. 


Figure  10.  Polarizabilities  Associated  with  Anomaly  WK-68,  a  Piece  of  Frag. 

Here  the  initial  acquisition  shows  good  resolution  of  target  features;  however,  the  recollect  provides 

features  dominated  by  the  ground  response. 

Multiple  objects  did  not  appear  to  present  any  significant  problems  for  the  in-field  QC.  No  cases 
were  found  where  an  absence  of  multi-source  solvers  in  the  in-field  models  led  to  a  poor 
classification  of  a  target.  While  there  were  several  instances  where  multiple  sources  were  found 
within  the  search  area,  most  of  these  cases  included  a  number  of  small  debris  items.  For  TOI  as 
small  as  20  mm,  it  is  possible  these  cases  could  have  been  problematic;  however,  the  smallest  TOI 
encountered  during  the  survey  was  37  mm,  which  was  not  small  enough  to  be  masked  by  any  of 
the  nearby  debris  items. 
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Out  of  the  231  recollects,  153  cases  were  found  where  the  soil  response  appeared  to  influence  the 
in-field  QC  decision  resulting  in  insufficient  characterization  of  the  area.  This  number  is 
significantly  greater  than  the  objective  5%  of  total  recollects;  however,  given  the  known  challenges 
associated  with  the  site’s  geology,  and  the  absence  of  any  methods  to  mitigate  these  effects  in  the 
in-field  software,  these  results  are  reasonable. 

7.4  OBJECTIVE:  EFFECTIVE  QUALITY  METRICS 

The  remainder  of  ineffective  recollects  (i.e.,  recollects  that  did  not  improve  characterization  and 
that  were  not  a  result  of  magnetic  geology  or  multiple  target  influence)  provided  an  opportunity  to 
test  quality  metrics  other  than  estimated  location  to  see  if  these  unnecessary  recollects  could  have 
been  avoided  by  using  additional  metrics.  Overall,  32  cases  were  found  where  the  initial 
acquisition  provided  sufficient  characterization  and  the  recollect  did  not  offer  any  significant 
improvements.  Figure  1 1  and  Figure  12  show  two  examples  (anomalies  WK-267  and  WK-169)  of 
such  cases. 


Figure  II.  Polarizabilities  Recovered  from  the  Initial  Acquisition  Data  (Left)  and  the 

Recollect  Data  (Right). 

These  polarizabilities  correspond  to  anomaly  WK-267,  a  medium  ISO  seed.  Here  the  recollect  was 

unnecessary. 
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Figure  12.  Polarizabilities  Recovered  from  the  Initial  Acquisition  Data  (Left)  and  the 

Recollect  Data  (Right). 

These  polarizabilities  correspond  to  anomaly  WK-169,  a  piece  of  frag.  Here  the  recollect  was 

unnecessary. 

For  several  of  these  eases,  the  initial  acquisition  was  very  close  to  the  30-cm  offset  threshold  used 
to  make  a  recollect  decision  (e.g.,  32  cm  for  WK-267,  and  26  cm  for  WK-169).  For  cases  like  these 
where  the  anomaly  is  close  to  the  offset  threshold,  including  a  data/model  fit  metric  into  the 
decision  could  boost  operator  confidence.  This  information  is  currently  provided  to  the  operator 
and  it  could  be  factored  into  the  decision.  For  example,  in  both  the  WK-267  and  WK-169  initial 
encounters,  the  data/model  fit  was  very  high  (>98%).  If  the  initial  acquisition  is  within,  for 
example,  25^0  cm  of  the  anomaly,  perhaps  a  fit  >98%  could  lead  to  a  no-recollect  decision. 

While  the  fit  could  be  a  good  supportive  metric  in  these  cases,  it  is  not  always  reliable.  Consider, 
for  example,  anomaly  WK-880,  a  fuze  clip  (Figure  13).  Both  the  initial  encounter  (37-cm  offset) 
and  the  recollect  (9-cm  offset)  provided  very  good  fits  (>98%  each);  however,  it  is  clear  that  the 
third  polarizability  (green  line  in  Figure  13)  is  not  well  constrained  in  the  initial  encounter  data 
(although  it  could  be  argued  that  these  initial  encounter  features  are  sufficient  for  making  a 
classification  decision).  In  this  case,  a  model  noise  metric  (i.e.,  polarizability  noise)  could  be  added 
to  the  decision  flow  to  determine  whether  a  recollect  is  necessary.  In  this  case,  the  initial  set  of 
polarizabilities  does  not  pass  the  model  noise  metric,  leading  to  a  recollect  decision. 
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Figure  13.  Polarizabilities  Recovered  from  the  Initial  Acquisition  Data  (Left)  and  the 

Recollect  Data  (Right). 

These  polarizabilities  correspond  to  anomaly  WK-880,  a  fuze  clip.  The  initial  data  do  not  fully  constrain 
the  third  polarizability  (green  line)  and  indicate  that  the  recollect  was  beneficial. 

Model  fit  provides  a  good  indication  that  the  target  is  well  represented  by  the  model  parameters. 
For  example,  cases  where  there  are  interfering  sources,  such  as  ground  response  or  other  nearby 
targets,  will  produce  a  reduced  fit  value.  Obtaining  a  high  model  fit  (i.e.,  >98%),  however,  does 
not  necessarily  indicate  that  the  model  parameters  are  well  constrained.  The  model  noise  metric  is 
a  measure  of  the  polarizability  stability  and  is  an  indication  of  whether  or  not  the  data  provide 
good  constraint  on  the  model  parameters.  For  example,  the  initial  features  shown  in  Figure  8  and 
Figure  13  indicate  that  the  third  polarizability  (green  line)  is  unstable  and  is,  therefore,  not  well 
constrained.  Taken  together,  these  two  metrics  could  be  a  robust  way  to  assess  whether  the  initial 
acquisition  data  provide  accurate  classification  features. 

One  possible  solution  for  reducing  unnecessary  recollects  is  to  include  these  additional  metrics 
into  the  decision  flow.  This  information  is  currently  available  to  the  operator;  however,  it  could  be 
possible  to  make  this  flow  automated  to  provide  a  recollect/no  recollect  decision  for  the  operator. 
Figure  14  presents  a  possible  flow  for  including  additional  metrics.  While  reducing  unnecessary 
recollects  is  not  critical  to  improving  the  outcome  of  the  in-field  QC  process,  it  could  improve  the 
efficiency  of  the  process. 
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Figure  14.  Possible  Decision  Flow  for  In-Field  QC. 

This  decision  flow  incorporates  the  additional  fit  and  model  noise  metrics  along  with  the  primary  target 
location  metric  to  support  the  decision  to  recollect  or  not.  The  objective  of  this  type  of  decision  flow 
would  be  to  reduce  unnecessary  recollects  that  may  occur  within  some  margin  of  error  of  the  30-cni  offset 
threshold  (e.g.,  25^0  cm).  All  metrics  including  the  offset  threshold  could  be  adjusted  for  site-specific 
requirements,  such  as  smallest  TOI  or  target  density,  to  reflect  different  tolerances  for  data  recollects. 

Out  of  the  32  cases  where  the  recollect  did  not  improve  target  features,  1 1  cases  were  found  that 
could  have  potentially  been  avoided  by  incorporating  the  basic  fit  and  model  noise  metrics  into 
the  decision  flow.  In  the  retrospective  analysis,  the  fit  metric  threshold  was  set  to  a  conservative 
value  (>98%)  such  that  these  additional  metrics  could  be  implemented  without  creating  missed 
recollect  opportunities.  Setting  a  high  value  for  the  fit  metric  was  an  effective  way  to  avoid  missed 
recollect  opportunities  as  most  of  the  “necessary”  recollect  cases  analyzed  failed  the  initial  fit 
metric  >98%. 

7.5  OBJECTIVE:  PRODUCTION  RATE 

For  the  production  rate  analysis,  the  project  team  wanted  to  establish  representative  production  rates 
for  surveys  performed  with  and  without  the  in-field  QC.  Factors  specific  to  the  WMA  survey,  such 
as  equipment  down-time  and  equipment  transport  time,  were  not  wanted  to  influence  these  rates. 
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Therefore,  the  project  team  relied  on  the  time  stamp  information  from  groups  of  consecutively 
recorded  data  files  that  were  created  during  periods  of  uninterrupted  survey  time.  The  time  stamp 
analysis  indicated  that  with  the  in-field  QC  process,  the  field  team  could  achieve  an  ideal  baseline 
production  rate  of  23  unique  anomalies/hr.  Without  the  in-field  QC  process  in  place,  the  field  team 
could  achieve  an  ideal  baseline  production  rate  of  27  unique  anomalies/hr. 

When  these  production  rates  were  adjusted  based  on  the  number  of  necessary  recollects  for  the 
survey,  the  true  production  rate  for  the  in-field  QC  approach  remained  at  23  anomalies/hr  while 
the  standard  approach  true  production  rate  dropped  to  26  anomalies/hr. 

Using  this  analysis,  it  appears  that  performing  a  survey  without  in-field  decision-based  recollects 
may  have  been  slightly  more  efficient  than  applying  the  in-field  QC  approach;  however,  a  number 
of  additional  factors  should  be  considered.  First,  a  large  number  of  recollects  (about  66%)  appeared 
to  be  a  result  of  ground  response  (i.e.,  no  contacts)  for  which  there  is  currently  no  in-field 
mitigation  process.  This  large  number  of  unnecessary  recollects  lowered  the  average  baseline 
production  rate  for  the  in-field  QC  survey.  Second,  the  number  of  necessary  recollects  (those  that 
led  to  improved  target  features)  was  relatively  small  at  46.  This  number  did  not  significantly  lower 
the  adjusted  rate  for  the  standard  approach  (it  dropped  from  27/hr  to  26/hr).  Therefore,  it  might  be 
expected  that  sites  without  difficult  geology  or  sites  that  contain  higher  anomaly  densities  could 
potentially  alter  these  production  rates  significantly. 

Another  way  to  assess  the  impact  of  the  in-field  QC  method  on  production  rate  is  to  view  it 
specifically  in  the  context  of  the  WMA  survey.  The  actual  production  rates  achieved  during  the 
survey  were  lower  than  the  23/hr  rate  would  indicate.  This  lower  rate  was  due  primarily  to  the 
difficult  terrain,  which  slowed  transport  of  the  sensor  across  larger  areas  of  the  site  (Van  et  ah, 
2015).  Actual  production  rates  achieved  were  in  the  range  of  65-75  anomalies/day.  Based  on  the 
data  time  stamp  information,  it  is  estimated  that  the  average  time  required  to  perform  a  recollect 
was  about  105  seconds.  Viewed  in  this  context,  the  231  recollects  probably  added  about  6-7  hrs 
of  additional  survey  time.  Going  back  and  reacquiring  46  anomalies  (the  necessary  recollects) 
would  have  probably  taken  1-2  days  of  additional  survey  time  given  that  these  anomalies  were 
spread  across  different  AOIs  within  the  site.  Thus,  it  is  likely  that  even  with  the  recollects 
associated  with  ground  response,  the  in-field  QC  enabled  a  slightly  more  efficient  survey  for  this 
particular  site. 

7.6  QUALITATIVE  OBJECTIVES 

Ease-of-use  has  been  discussed  with  a  number  of  MetalMapper  system  operators  who  have 
performed  surveys  with  the  in-field  QC  software.  Overall,  feedback  has  been  positive.  One  of  the 
most  common  reports  is  that  having  an  inversion-based  location  that  can  be  used  to  position  the 
sensor  provides  the  operator  with  additional  confidence  in  the  quality  of  the  data.  Another  positive 
response  has  been  utility  of  the  software  for  performing  Instrument  Verification  Strip  (IVS) 
activities.  Having  the  in-field  output  for  the  polarizabilities  and  estimated  location  of  each  IVS 
item  provides  immediate  and  quantitative  feedback  about  the  functionality  of  the  system  hardware. 
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8.0  COST  ASSESSMENT 


The  MetalMapper  survey  eosts  can  be  found  in  the  MR-201104  final  report  (Van  et  ah,  2015). 
Total  costs  applying  the  in-field  QC  approach  during  the  survey  were  $76,100  for  cued  acquisition 
of  1,032  unique  anomalies.  This  figure  produces  an  average  rate  of  $74/anomaly.  This  rate  can  be 
adjusted  for  the  standard  method  (i.e.,  no  in-field  recollects)  using  the  aforementioned  true 
production  rates  for  both  methods.  An  additional  cost  component  is  the  time  required  for  set  up 
and  installation  of  the  in-field  QC  software. 

8.1  COST  DRIVERS 

Production  rate  analysis  indicates  that  the  cost  per  anomaly  might  have  been  lower  if  the  survey 
had  been  applied  without  recollects,  due  to  the  fact  that  a  large  number  of  recollects  (153)  were 
associated  with  ground  response,  which  produced  inaccurate  source  locations.  Additionally,  there 
were  relatively  few  recollects  (46  out  of  23 1  total)  that  provided  significant  improvements  in  target 
features.  These  cost  drivers  reflect  the  challenges  that  were  specific  to  the  WMA  site. 

The  true  cost  comparison  of  the  two  methods  for  the  WMA  site  may  be  somewhat  different  than 
what  is  reflected  by  the  cost  per  anomaly  in  Table  2  if  the  specific  conditions  of  the  site  are 
considered.  For  example,  it  is  unlikely  that  reacquisition  of  46  targets  could  actually  be 
accomplished  in  ~2  hrs.  In  reality,  this  would  likely  have  required  1-2  additional  survey  days  given 
that  these  anomalies  covered  three  different  AOls  and  transport  between  these  areas  was  time- 
consuming.  It  is  expected  that  the  true  costs  for  each  method  at  this  particular  site  were  comparable, 
if  not  slightly  lower,  for  the  in-field  QC  method. 

Table  2.  Cost  Model  for  In-Field  QC  for  Cued  EMI  Surveys. 


Approach 

Data  to  be  Tracked 

Survey  costs  (in-field  QC) 

$74/anomaly 

•  Assumes  1,032  unique  anomalies  surveyed  at  a  rate  of  23  unique 
anomalies/hr 

•  Ineludes  153  reeolleets  assoeiated  with  ground  response 

Survey  costs  (standard) 

$65/anomaly 

•  Assumes  1,032  unique  anomalies  surveyed  at  a  rate  of  26  unique 
anomalies/hr 

•  Ineludes  46  reaequisitions 

T  raining/installation 

$2,400 

•  Approximately  Vi  day  of  survey  time 
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9.0  IMPLEMENTATION  ISSUES 


The  greatest  ehallenge  to  effective  implementation  of  the  in-field  QC  process  at  the  WMA  site  was 
the  presence  of  significant  magnetic  geology  throughout  the  AOIs.  The  retrospective  analysis  of 
the  survey  data  indicated  that  a  significant  portion  of  recollects  based  on  the  in-field  decision  were 
a  result  of  the  soil  response  (153  out  of  231).  After  analyzing  the  recollect  data,  the  following 
recommendations  were  developed  that  could  improve  the  effectiveness  and  efficiency  of  the  in¬ 
field  QC  process  at  other  challenging  sites: 

•  One  approach  to  enable  in-field  mitigation  of  magnetic  geology  could  be  to  enhance  the 
background  selection  and  removal  process  of  the  in-field  software.  For  example,  multiple 
background  locations  based  on  DGM  analysis  of  magnetic  geology  in  the  survey  area  could 
be  included  in  the  background  selection  process  for  the  in-field  software.  Background  files 
collected  at  these  locations  could  be  loaded  into  the  module  and  automatically  selected 
based  on  location  of  the  sensor.  Proximity  to  different  background  locations  in  the  site 
would  drive  the  selection  of  the  optimal  background  file  to  apply  before  inverting  each 
subsequent  data  file. 

•  In  many  cases,  however,  there  will  not  be  an  ideal  background  file.  This  was  evident  in  the 
post-survey  processing  when  it  became  apparent  that  none  of  the  background  files  collected 
on  a  given  day  were  adequate  for  removing  the  ground  response  for  a  number  of  anomaly 
encounters.  In  cases  such  as  these,  more  sophisticated  inversion  strategies  are  required.  For 
example,  replicating  the  ground  response  by  constraining  the  location  of  a  deep  dipole 
source  during  inversion  has  been  effective  for  isolating  the  response  from  targets  buried  in 
magnetic  soil  (e.g.,  Pasion  et  ah,  2012).  It  could  be  an  effective  approach  to  include  a  two- 
source  solver  in  the  in-field  model  that  would  allow  one  source  to  be  constrained  to 
represent  the  ground  response.  Decay  fit  metrics  could  be  applied  to  determine  if  the 
constrained  source  does  indeed  produce  ground-like  features  that  indicate  the  presence  of 
significant  magnetic  geology. 

•  While  the  WMA  survey  did  not  appear  to  produce  many  cases  where  multiple  objects 
influenced  the  in-field  QC  decision,  there  are  sites  where  this  could  be  an  issue.  In  some 
cases,  implementing  a  multi-source  solver  could  be  effective  for  isolating  the  ground 
response  (see  above)  as  well  as  for  isolating  responses  from  multiple,  closely-spaced 
targets. 

•  Finally,  using  quality  metrics  in  addition  to  the  estimated  source  location  could  prove 
useful  for  eliminating  unnecessary  recollects.  In  cases  where  the  target  is  isolated  from 
any  interfering  sources  (e.g.,  ground  response,  other  nearby  targets,  low  signal-to-noise 
ratio),  the  30-cm  offset  threshold  may  be  overly  stringent.  While  eliminating  unnecessary 
recollects  may  not  be  critical  to  improving  the  effectiveness  of  the  process,  it  could 
improve  the  efficiency  of  the  process,  leading  to  better  overall  cost  performance. 
Incorporating  other  metrics  into  the  decision  flow  (Figure  14,  for  example)  could 
eliminate  recollects  in  some  of  these  cases  where  the  initial  acquisition  provides  adequate 
characterization  of  the  target. 
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While  the  WMA  survey  elueidated  several  ehallenges  for  the  in-field  QC  proeess,  it  is  believed 
that  overall  the  proeess  proved  benefieial  for  the  survey.  There  were  46  eases  where  the  reeolleet 
provided  signifieant  improvements  in  the  eharaeterization  of  the  target  and  most  likely  improved 
the  final  elassifieation  result.  Given  the  ehallenges  of  transporting  and  moving  equipment  around 
the  different  AOIs,  these  46  anomalies  would  likely  have  added  an  additional  1-2  days  of 
surveying  if  they  had  not  been  reeolleeted  based  on  the  in-field  deeision  (for  eomparison,  it  is 
estimated  that  the  23 1  total  reeolleets  for  the  survey  added  6-7  hrs  of  aetual  survey  time  over  the 
eourse  of  field  aetivities).  Additionally,  there  are  other  benefits  to  providing  the  field  team  with 
immediate  inversion-based  metries  ineluding  the  immediate  feedbaek  about  instrument 
funetionality  during  IVS  aetivities. 

It  should  also  be  noted  that  during  most  of  the  survey,  one  of  the  eomer  reeeivers  on  the 
MetalMapper  was  malfunetioning.  It  is  unelear  to  what  extent  this  faulty  reeeiver  influeneed  the 
in-field  deeisions;  however,  it  is  possible  that  it  had  some  effeet  on  the  aeeuraey  of  in-field  target 
loeation  estimates. 

It  is  likely  that  a  large  number  of  sites  eould  benefit  from  implementing  an  in-field  QC  proeess  for 
cued  surveys.  Applying  some  of  the  aforementioned  improvements  could  make  the  process  more 
effective  for  particularly  challenging  sites,  such  as  those  containing  magnetic  geology  or  high 
anomaly  densities. 
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